DataFrame Schema & Metadata Operations

Inspecting DataFrame structural schemas and generating descriptive summaries in PySpark using printSchema, describe, and summary.

What are Schema & Metadata Operations?

Understanding the structure and distribution of data inside a DataFrame is the initial phase of any data science or ETL process. PySpark provides built-in operations to inspect and describe DataFrames:

• printSchema(): Displays the nested tree structure of column names, data types, and nullability.
• describe(colNames...): Calculates basic statistical metrics (count, mean, stddev, min, max) for numeric/string columns.
• summary(statistics...): Calculates granular descriptive statistics, including explicit quartiles and percentiles.

Syntax and Core Inspections

# A. Inspect structural schema (printed instantly to stdout)
df.printSchema()

# B. Get list of column names
column_names = df.columns

# C. Get list of column names paired with their data type
column_dtypes = df.dtypes

# D. Basic Statistics for specific columns
df.describe("salary", "age").show()

# E. Custom percentiles using summary
df.summary("count", "mean", "25%", "50%", "75%", "max").show()

Example Usage Pipeline

Below is a complete, copy-paste-ready PySpark script demonstrating schema and summary inspections:

from pyspark.sql import SparkSession

# 1. Setup local Spark session
spark = SparkSession.builder \
    .appName("DataFrame Metadata Demo") \
    .master("local[*]") \
    .getOrCreate()

# 2. Dummy dataset (Finances)
data = [
    ("Alice", 28, 92000.0),
    ("Bob", 34, 61000.0),
    ("Charlie", 45, 95000.0),
    ("David", 22, 50000.0),
]
columns = ["name", "age", "salary"]
df = spark.createDataFrame(data, columns)

# 3. Print structural metadata
print("=== Structural Schema printSchema() ===")
df.printSchema()

# 4. Compute basic descriptive statistics
print("=== basic describe() Output ===")
df.describe("age", "salary").show()

# 5. Compute advanced summary with custom percentiles
print("=== Advanced summary() Output ===")
df.select("age", "salary").summary("count", "mean", "50%", "max").show()

Rendered Output:

=== Structural Schema printSchema() ===
root
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)
 |-- salary: double (nullable = true)

=== basic describe() Output ===
+-------+------------------+-----------------+
|summary|               age|           salary|
+-------+------------------+-----------------+
|  count|                 4|                4|
|   mean|             32.25|          74500.0|
| stddev|9.776672917377038 |22037.84623475141|
|    min|                22|          50000.0|
|    max|                45|          95000.0|
+-------+------------------+-----------------+

=== Advanced summary() Output ===
+-------+-----+-------+
|summary|  age| salary|
+-------+-----+-------+
|  count|    4|      4|
|   mean|32.25|74500.0|
|    50%|   31|61000.0|
|    max|   45|95000.0|
+-------+-----+-------+